Vehicle sensor with integrated radar and image sensors

ABSTRACT

A sensing system of a vehicle includes a sensor module disposed at the vehicle. The sensor module includes first and second radar sensors and a camera. A field of sensing of the first radar sensor is encompassed by a portion of a field of view of the camera and a field of sensing of the second radar sensor is encompassed by another portion of the field of view of the camera. Outputs of the radar sensors and the camera are communicated to a control. The control, responsive to processing of the outputs of the radar sensors, detects the presence of an object exterior the vehicle and within the field of sensing of at least one of the radar sensors. The control, responsive to detection of an object via processing of the outputs of the radar sensors, processes the output of the camera to classify the detected object.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims the filing benefits of U.S. provisionalapplications, Ser. No. 62/398,094, filed Sep. 22, 2016, and Ser. No.62/378,849, filed Aug. 24, 2016, which are hereby incorporated herein byreference in their entireties.

FIELD OF THE INVENTION

The present invention relates generally to a vehicle sensing system fora vehicle and, more particularly, to a vehicle sensing system thatutilizes one or more sensors at a vehicle to provide a field of sensingat or around the vehicle.

BACKGROUND OF THE INVENTION

Use of imaging sensors or ultrasonic sensors or radar sensors in vehiclesensing systems is common and known. Examples of such known systems aredescribed in U.S. Pat. Nos. 8,013,780 and 5,949,331 and/or U.S.publication No. US-2010-0245066 and/or International Publication No. WO2011/090484, which are hereby incorporated herein by reference in theirentireties.

SUMMARY OF THE INVENTION

The present invention provides a driver assistance system or sensingsystem for a vehicle that utilizes a sensor module disposed at thevehicle to sense a respective region exterior of the vehicle, with thesensor module comprising one or more radar sensors and at least onecamera disposed in a common housing. A field of sensing of the radarsensor(s) is encompassed by a portion of a field of view of the at leastone camera. The system includes a control, where outputs of the radarsensor(s) and the at least one camera are communicated to the control,and where the control, responsive to processing of the outputs of theradar sensor(s), detects the presence of one or more objects exteriorthe vehicle and within the field of sensing of at least one of at leastone radar sensor, and where the control, responsive to processing of theoutput of the at least one camera, classifies the detected object.

According to an aspect of the present invention, a sensing system of avehicle includes a sensor module disposed at a vehicle. The sensormodule includes first and second radar sensors and a camera. A field ofsensing of the first radar sensor is encompassed by a portion of a fieldof view of the camera and a field of sensing of the second radar sensoris encompassed by another portion of the field of view of the camera.Outputs of the radar sensors and the camera are communicated to acontrol. The control, responsive to processing of the outputs of theradar sensors, detects the presence of an object exterior the vehicleand within the field of sensing of at least one of the radar sensors.The control, responsive to detection of an object via processing of theoutputs of the radar sensors, processes the output of the camera toclassify the detected object.

The sensor module may comprise a circuit board with the first and secondradar sensors and the camera disposed at the circuit board, and with thecamera disposed at the circuit board at a location between the first andsecond radar sensors. Physical orientation of the second radar sensor atthe circuit board relative to the camera may be different from physicalorientation of the first radar sensor at the circuit board relative tothe camera. For example, one may be oriented for horizontal resolutionand one may be oriented for vertical resolution (where they are orientedwith their antenna arrays rotated about 90 degrees relative to oneanother).

The sensor module may be disposed at a rear of the vehicle such that thecamera views rearward of the vehicle. During a reversing maneuver of thevehicle, a display device of the vehicle may display video imagesderived from the output of the camera.

These and other objects, advantages, purposes and features of thepresent invention will become apparent upon review of the followingspecification in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of a vehicle with a sensing system thatincorporates a radar sensor in accordance with the present invention;

FIG. 2 is a schematic showing the regions encompassed by the fields ofsensing of a pair of radar sensors and a camera;

FIG. 3 is a schematic showing the regions encompassed by the fields ofsensing of a radar and a camera disposed in a common housing inaccordance with the present invention;

FIG. 4 is a side elevation of a vehicle equipped with a sensing systemof the present invention, showing the pitch boresight of the radarsensor;

FIG. 5 is a rear elevation of the vehicle equipped with the sensingsystem, showing the roll boresight of the radar sensor;

FIG. 6 is a top plan view of the vehicle equipped with the sensingsystem, showing the yaw boresight of the radar sensor;

FIG. 7 is a schematic of an integrated camera and radar device, showingthe radar sensors at 90 degrees relative to one another; and

FIG. 8 is a schematic of an integrated camera and radar device, showingthe radar sensors at 70 degrees relative to one another.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

A vehicle sensing system, such as a driver or driving assist system,object detection system, parking assist system and/or alert system,operates to capture sensing data exterior of the vehicle and may processthe captured data to detect objects at or near the vehicle and in thepredicted path of the vehicle, such as to assist a driver of the vehiclein maneuvering the vehicle in a forward or rearward direction or toassist the driver in parking the vehicle in a parking space. The systemincludes a processor that is operable to receive sensing data frommultiple sensors and to provide an output to a control that, responsiveto the output, generates an alert or controls an accessory or system ofthe vehicle, or highlights or overlays an alert on a display screen(that may be displaying video images captured by a single rearwardviewing camera or multiple cameras providing forward, side or 360 degreesurround views of the area surrounding the vehicle during a reversing orlow speed maneuver of the vehicle).

Referring now to the drawings and the illustrative embodiments depictedtherein, a vehicle 10 includes an driver assistance system or sensingsystem 12 that includes at least one radar sensor unit, such as aforward facing radar sensor unit 14 (and the system may optionallyinclude multiple exterior facing sensors, such as cameras or othersensors, such as a rearward facing sensor at the rear of the vehicle,and a sideward/rearward facing sensor at respective sides of thevehicle), which sense regions exterior of the vehicle. The sensingsystem 12 includes a control or electronic control unit (ECU) orprocessor that is operable to process data captured by the sensor orsensors and may detect objects or the like. The data transfer or signalcommunication from the sensor to the ECU may comprise any suitable dataor communication link, such as a vehicle network bus or the like of theequipped vehicle.

Some automotive radars use MIMO (Multiple Input Multiple Output)techniques to create an effective virtual antenna aperture, which issignificantly larger than the real antenna aperture, and delivers muchbetter angular resolution than conventional radars, such as, forexample, conventional scanning radars. MIMO techniques may be used tocreate virtual antenna apertures, not only from linear arrays of realantennas but also from two dimensional arrays of real antennas. Forexample, the antenna array of a MIMO sensor may comprise twotransmitting antennas and two or more receiving antennas, arrangedeither in a one-dimensional array or as in a two-dimensional array.Thus, the antenna array may consist of four antennas (arranged as a 2×2array MIMO virtual antenna) or any other combination of transmitting andreceiving antennas, whose product is the number of virtual antennas,such as, for example, 16 virtual antennas (4×4 array) or more or lessthan 16 virtual antennas.

For vision systems having an exterior viewing camera, the challenge isthat the OEM system level requirements may specify that objects need tobe detected and classified between 0 m . . . 18 m with a resolution inthe 10 cm range. In order to ‘see’ enough of the object in the closerange (0 . . . 2 m), a camera lens with about a 185 degree opening angleis needed producing a fisheye image. The large opening angle of the lensin combination with an imager with limited resolution (such as around 1MP, which are spread in an angle to cover the desired wide field ofview) causes a very poor resolution in the long range. For example, at10 m range the pixel resolution is around 1.2 m, meaning that distancemeasurement with enough precision beyond 10 m is difficult to nearlyimpossible.

For a radar sensing system with a 16 virtual receiver configuration, theangular resolution is approximately 10 degrees. The spot size of a sliceof the radar's field of view would be approximated by the equation(angular resolution×range)/57. Thus, at one meter, the spot slice isabout 17 cm. As the range is extended, the spot size would increase.

If a radar sensor with horizontal slices and another radar sensor withvertical slices are utilized to cover an FOV, the intersection of theirslices one from each sensor permits a 17 square cm spot in 2-D for fineresolution over the common FOV. The BSD (blind spot detection) cornersensors would allow for conversion of the spot to a cube for threedimensional (3D) and terrain monitoring.

Corner radar sensors may be used for detection and cameras may be usedfor classification. For example, the radar sensors mounted at thecorners of the vehicle are used for the object detection and therearward viewing camera may be used for classification (and optionallyfor display of video images derived from and representative of imagedata captured by the rearward viewing camera). The radar sensors can beused for measurement of object height and object location. A higheraccuracy is achieved in the FOV where the radar sensors overlap.However, a radar blindspot may occur at the center area of the vehicle(see FIG. 2).

The present invention provides an integrated radar sensor and camerathat are integrated into the same or common housing (see FIG. 3). Theradar sensor is used for object detection and object localization, witha vertical opening angle of about 150 degrees, a range of about 30 m,and range resolution of about 7.5 cm. The radar sensor allows formeasuring of the height of objects, the relative velocity and path. Thecamera and machine vision/image processing is used for objectclassification (such as, for example, pedestrians, vehicles, bicycles,and/or the like).

For example, the radar and camera data may be fused for an automaticemergency braking (AEB) system of the vehicle. The machine vision/imageprocessing of the camera does not require object detection anymore,therefore is of less risk for smart camera implementations (lessprocessing performance required). If the camera and radar are packagedtogether, there is less integration effort for the OEM, also wiring isminimized.

Optionally, the system may integrate radar sensor(s) in stereo threedimensional (3D) configuration and a camera into a common housing. Theradar sensors are positioned such that their horizontal and verticalazimuths are intersecting. It is envisioned that the radar sensors havecommon horizontal and vertical fields of view or fields of sensing andcommon azimuth resolution. Each radar sensor is used for objectdetection and object localization, with a vertical opening angle ofabout 150 degrees, a range of about 30+m, and a range resolution of 7.5cm. The radar sensor allows measuring of the height of objects andcreation of a terrain map. The pitch of the radar sensor mounting (seeFIG. 4) is selected to maximize the coverage close to the vehicle, withthe vertical field of view/sensing ideally intersecting the vehicle. Theroll (FIG. 5) and yaw (FIG. 6) of the radar sensors is selected tobalance depth measurement across the entire combined field of sensing ofall of the radar sensors.

It is envisioned that the sensors' horizontal planes are rotated greaterthan about 90 degrees relative to each other, providing intersectingbeams in range, velocity, azimuth and elevation for all detections,creating radar data cubes of known size for all volumes in 3D spacewithin the combined field of view of at least three sensors on thevehicle. Camera, radar and machine vision/image processing is used forobject classification (such as, for example, pedestrians, vehicles,bicycles, and/or the like). The captured radar data and camera/imagedata is fused for automatic emergency braking (AEB) applications.

Optionally, the radar sensors may operate collaboratively, where thetransmitted signals from each transmit antenna of radar sensor #1 arereceived and processed by all receive antenna of both radar sensor #1and radar sensor #2, thereby increasing effective aperture, accuracy andresolution of the system.

The machine vision/image processing of the camera-captured image datadoes not require object detection, and thus is of less risk for smartcamera implementations (less processing performance required). If thecamera and radar are packaged together, there is less integration effortfor the OEM, while also minimizing or reducing wiring requirements.

As shown in FIGS. 7 and 8, an integrated camera and radar package mayinclude a camera disposed between two radar sensors. Optionally, theradar sensors may be arranged at 90 degrees relative to one another(FIG. 7), such that (when the device is mounted at a vehicle) one radarsensor antenna is at a vertical resolution orientation and the otherradar sensor antenna is at a horizontal resolution orientation, thusproviding a symmetric radar cube package. Optionally, the radar sensorsmay be arranged at a different angle (such as, for example, 70 degreesin FIG. 8) relative to one another, such that (when the device ismounted at a vehicle) one radar sensor antenna is at a roll+35 degreeorientation and the other radar sensor antenna is at a roll−35 degreeorientation, thus providing an asymmetric radar cube package. Clearly,other orientations may be implemented depending on the particularapplication of the integrated radar and camera sensing device andsystem.

The system may utilize sensors, such as radar or lidar sensors or thelike. The sensing system may utilize aspects of the systems described inU.S. Pat. Nos. 9,753,121; 9,689,967; 9,599,702; 9,575,160; 9,146,898;9,036,026; 8,027,029; 8,013,780; 6,825,455; 7,053,357; 7,408,627;7,405,812; 7,379,163; 7,379,100; 7,375,803; 7,352,454; 7,340,077;7,321,111; 7,310,431; 7,283,213; 7,212,663; 7,203,356; 7,176,438;7,157,685; 6,919,549; 6,906,793; 6,876,775; 6,710,770; 6,690,354;6,678,039; 6,674,895 and/or 6,587,186, and/or International PublicationNo. WO 2011/090484 and/or U.S. Publication Nos. US-2017-0222311 and/orUS-2010-0245066, and/or U.S. patent application Ser. No. 15/647,339,filed Jul. 12, 2017 (Attorney Docket MAG04 P-3071 R), Ser. No.15/619,627, filed Jun. 12, 2017 (Attorney Docket MAG04 P-3056R), Ser.No. 15/584,265, filed May 2, 2017 (Attorney Docket MAG04 P-3017R), Ser.No. 15/467,247, filed Mar. 23, 2017 (Attorney Docket MAG04 P-2978R),Ser. No. 15/446,220, filed Mar. 1, 2017 (Attorney Docket MAG04 P-2955),and/or Ser. No. 15/675,919, filed Aug. 14, 2017 (Attorney Docket MAG04P-3092R), and/or International PCT Application No. PCT/IB2017/054120,filed Jul. 7, 2017 (Attorney Docket MAG04 FP-3069PCT), which are herebyincorporated herein by reference in their entireties.

Changes and modifications in the specifically described embodiments canbe carried out without departing from the principles of the invention,which is intended to be limited only by the scope of the appendedclaims, as interpreted according to the principles of patent lawincluding the doctrine of equivalents.

1. A sensing system of a vehicle, said sensing system comprising: asensor module disposed at a vehicle equipped with said sensing system;wherein said sensor module comprises (i) a first radar sensor having aplurality of transmitting antennas and a plurality of receivingantennas, (ii) a second radar sensor having a plurality of transmittingantennas and a plurality of receiving antennas, and (iii) a camera;wherein said first radar sensor has a first field of sensing exteriorthe equipped vehicle, and wherein said second radar sensor has a secondfield of sensing exterior the equipped vehicle, and wherein said camerahas a field of view exterior the equipped vehicle; wherein the firstfield of sensing of said first radar sensor is encompassed by a portionof the field of view of said camera, and wherein the second field ofsensing of said second radar sensor is encompassed by another portion ofthe field of view of said camera; a control, wherein outputs of saidfirst and second radar sensors and said camera are communicated to saidcontrol; wherein said control comprises a processor that processes theoutputs; wherein said control, responsive to processing of the outputsof said first and second radar sensors, detects the presence of anobject exterior the vehicle and within the field of sensing of at leastone of said first radar sensor and said second radar sensor; and whereinsaid control, responsive to detection of the presence of an object viaprocessing of the outputs of said first and second radar sensors,processes the output of said camera to classify the detected object. 2.The sensing system of claim 1, wherein physical orientation of saidsecond radar sensor relative to said camera is different from physicalorientation of said first radar sensor relative to said camera.
 3. Thesensing system of claim 1, wherein said first and second radar sensorsare positioned such that their horizontal and vertical azimuths areintersecting.
 4. The sensing system of claim 1, wherein said sensormodule comprises a circuit board and wherein said first and second radarsensors and said camera are disposed at said circuit board, and whereinsaid camera is disposed at said circuit board at a location between saidfirst and second radar sensors.
 5. The sensing system of claim 1,wherein said first and second radar sensors and said camera are disposedin a common housing.
 6. The sensing system of claim 1, wherein saidsensor module is disposed at a rear of the vehicle such that said cameraviews rearward of the vehicle.
 7. The sensing system of claim 6,wherein, during a reversing maneuver of the vehicle, a display device ofthe vehicle displays video images derived from the output of saidcamera.
 8. The sensing system of claim 1, wherein a display device ofthe vehicle is operable to display video images derived from the outputof said camera.
 9. The sensing system of claim 1, wherein each of saidfirst and second radar sensors comprises an array of virtual antennas.10. The sensing system of claim 9, wherein said array of virtualantennas comprises at least four virtual antennas.
 11. The sensingsystem of claim 1, wherein said sensor module is part of a sensingsystem capable of providing driver assist system functions.
 12. Thesensing system of claim 11, wherein said sensing system providesdetection for at least one of automated parking, blind spot detection,cross traffic alert, lane change and merge aid, automatic emergencybraking, pedestrian detection, turn assist, terrain mapping andintersection collision mitigation.
 13. The sensing system of claim 1,wherein radar signals transmitted by said plurality of transmittingantennas of said first radar sensor are received by said plurality ofreceiving antennas of said second radar sensor, and wherein radarsignals transmitted by said plurality of transmitting antennas of saidsecond radar sensor are received by said plurality of receiving antennasof said first radar sensor.
 14. A sensing system of a vehicle, saidsensing system comprising: a sensor module disposed at a vehicleequipped with said sensing system; wherein said sensor module comprises(i) a first radar sensor having a plurality of transmitting antennas anda plurality of receiving antennas, (ii) a second radar sensor having aplurality of transmitting antennas and a plurality of receivingantennas, and (iii) a camera; wherein said sensor module comprises acircuit board and wherein said first and second radar sensors and saidcamera are disposed at said circuit board, and wherein said camera isdisposed at said circuit board at a location between said first andsecond radar sensors; wherein physical orientation of said second radarsensor at said circuit board relative to said camera is different fromphysical orientation of said first radar sensor at said circuit boardrelative to said camera; wherein said first radar sensor has a firstfield of sensing exterior the equipped vehicle, and wherein said secondradar sensor has a second field of sensing exterior the equippedvehicle, and wherein said camera has a field of view exterior theequipped vehicle; wherein the first field of sensing of said first radarsensor is encompassed by a portion of the field of view of said camera,and wherein the second field of sensing of said second radar sensor isencompassed by another portion of the field of view of said camera; acontrol, wherein outputs of said first and second radar sensors and saidcamera are communicated to said control; wherein said control comprisesa processor that processes the outputs; wherein said control, responsiveto processing of the outputs of said first and second radar sensors,detects the presence of an object exterior the vehicle and within thefield of sensing of at least one of said first radar sensor and saidsecond radar sensor; and wherein said control, responsive to detectionof the presence of an object via processing of the outputs of said firstand second radar sensors, processes the output of said camera toclassify the detected object.
 15. The sensing system of claim 14,wherein said first and second radar sensors are positioned such thattheir horizontal and vertical azimuths are intersecting.
 16. The sensingsystem of claim 14, wherein said first and second radar sensors and saidcamera are disposed in a common housing.
 17. The sensing system of claim14, wherein radar signals transmitted by said plurality of transmittingantennas of said first radar sensor are received by said plurality ofreceiving antennas of said second radar sensor, and wherein radarsignals transmitted by said plurality of transmitting antennas of saidsecond radar sensor are received by said plurality of receiving antennasof said first radar sensor.
 18. A sensing system of a vehicle, saidsensing system comprising: a sensor module disposed at a vehicleequipped with said sensing system; wherein said sensor module comprises(i) a first radar sensor having a plurality of transmitting antennas anda plurality of receiving antennas, (ii) a second radar sensor having aplurality of transmitting antennas and a plurality of receivingantennas, and (iii) a camera; wherein said sensor module comprises acircuit board and wherein said first and second radar sensors and saidcamera are disposed at said circuit board, and wherein said camera isdisposed at said circuit board at a location between said first andsecond radar sensors; wherein said first radar sensor has a first fieldof sensing exterior the equipped vehicle, and wherein said second radarsensor has a second field of sensing exterior the equipped vehicle, andwherein said camera has a field of view exterior the equipped vehicle;wherein the first field of sensing of said first radar sensor isencompassed by a portion of the field of view of said camera, andwherein the second field of sensing of said second radar sensor isencompassed by another portion of the field of view of said camera;wherein said sensor module is disposed at a rear of the vehicle suchthat said camera views rearward of the vehicle; a control, whereinoutputs of said first and second radar sensors and said camera arecommunicated to said control; wherein said control comprises a processorthat processes the outputs; wherein said control, responsive toprocessing of the outputs of said first and second radar sensors,detects the presence of an object exterior the vehicle and within thefield of sensing of at least one of said first radar sensor and saidsecond radar sensor; wherein said control, responsive to detection ofthe presence of an object via processing of the outputs of said firstand second radar sensors, processes the output of said camera toclassify the detected object; and wherein, during a reversing maneuverof the vehicle, a display device of the vehicle displays video imagesderived from the output of said camera.
 19. The sensing system of claim18, wherein said first and second radar sensors are positioned such thattheir horizontal and vertical azimuths are intersecting.
 20. The sensingsystem of claim 18, wherein radar signals transmitted by said pluralityof transmitting antennas of said first radar sensor are received by saidplurality of receiving antennas of said second radar sensor, and whereinradar signals transmitted by said plurality of transmitting antennas ofsaid second radar sensor are received by said plurality of receivingantennas of said first radar sensor.